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Abstract 

We present version 4.0 of the symbohc manipulation system Form. The most important 
new features are manipulation of rational polynomials and the factorization of expressions. 
Many other new functions and commands are also added; some of them are very general, 
while others are designed for building specific high level packages, such as one for Grobner 
;_^ ', bases. New is also the checkpoint facility, that allows for periodic backups during long 

calculations. Lastly, Form 4.0 has become available as open source under the GNU General 
Public License version 3. 



1 Introduction 

Over the years the symbohc manipulation system FORM p[] has undergone many changes. 
When its first version was released in 1989, it was capable of dealing with the problems that 
existed at that time: taking traces of strings of Dirac gamma matrices, executing operations 
on symbols and dot products of vectors, and manipulating functions with simple arguments. 
As a successor of Schoonschip [2] Form went in most of these properties just a bit beyond 
its predecessor's capabilities. Its main features were already its speed and its facilities for 
very large expressions. 

Over the years the computations done with Form became more and more complex and 
hence the program was extended regularly. One of the focuses of development has been the 
dealing with functions, their argument fields and pattern matching. More communication 
between the contents of the expressions and the input program has been provided with 
introducing the $-variables in version 3. This version also contained a complete rewriting 
of the system that processes the input, which made it much easier to add new commands. 
In addition the language became a bit more coherent. Form version 3 was the result of 11 
years of experience with different users doing calculations and this had produced a system 
with many original features. However, one of the capabilities that was missed by many users 
was the factorization of expressions. This has been added in version 4. Another feature 
that was requested by many users was the ability to work with rational polynomials as 
coefficients of terms. In the past this problem was addressed by letting Form interact with 
external programs [3] and send such polynomial work to other programs like Fermat [1] . After 
the external program has finished its work. Form would receive the results and proceeds 
from there. In version 4 this is no longer necessary for rational polynomial arithmetic. 
Nonetheless, the external communications facility still remains and can be used for many 
other things. 

Another feature, that has been added over the years, was running Form on more than 
a single processor. This started with the ParFoRM [5] project in Karlsruhe. ParFoRM is 
developed to run across different computers over a network using MPI, but it can also be 
used on computers with multiple cores. For this last type of computers also a multithreaded 
version of FoRM was created, named TFoRM [6]. In TFoRM most features were easier to 
implement. The disadvantage of ParFoRM is that it has more communication overhead. The 
disadvantage of TFoRM when compared to ParFoRM is that TFoRM can only run on single 
multicore computers. When all workers have a heavy interaction with the (usually single) 
disk, there may be a severe slowdown due to a traffic jam at the disk. When ParFoRM runs 
over a network on several computers, each with their own disk, this problem does not occur. 
For both ParFoRM and TFoRM the computing model is such that most programs that are 
created for regular (sequential) FORM, will not need any modification to benefit from the 
extra cores by running faster. At times some statements can be added to the program to 
improve the performance even further. 

One of the problems of developing a language is backward compatibility. The transition 
from version 2 to 3 saw a number of language features changed. To aid the users in the 
transition a conversion program was provided, but still there were cases in which manual 
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intervention was necessary. Also, some users had been using, against the instructions of the 
manual, a variety of bugs in the system that were repaired in version 3. As a result the 
conversion of old version 2 code was sometimes a bit painful. The introduction of version 4 
has none of these problems. All version 3 programs should work with version 4. The only 
things that have been changed in this respect are some default settings, but they can be 
restored to their old values in the setup file if wanted. 

With the introduction of version 4 some extra facilities become available. Form is now 
open source under the GNU General Public License [7j version 3 and there is a publicly 
accessible CVS repository from which anybody can download the sourcedj. In addition 
there is quite an amount of documentation. Another facility is the forunu on which users 
and developers can discuss with each other. This has already been very helpful in locating 
bugs and installation problems. 

The outline of this paper is as follows. In section 2 we discuss the generic structure of 
Form, TForm and ParFoRM. In section 3 we discuss Form being open source and address 
the problem of adding new features. Section 4 is dedicated to the new features that were 
introduced in the later editions of version 3 and in version 4. In section 5 details of the 
algorithms used for polynomial algebra are discussed. Some attention is given to the parallel 
versions in section 6. Section 7 addresses generic facilities and the conclusions are given in 
section 8. 

2 Generic structure 

Before going into the new features of version 4 it is best to consider first the internal structure 
of Form. The system consists of a number of modules that have a limited amount of 
interaction with each other, namely 

• the preprocessor, 

• the compiler, 

• the pattern matcher, 

• the terms generator, 

• the normalization routines, and 

• the sorting system. 

In addition there are routines that deal with specific tasks such as taking traces of Dirac 
gamma matrices, initialization, storage, and handling table bases. There are also libraries 
for many types of standard operations like arithmetic, writing files, and compression. When 
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available, Form uses libz [8] for the gzip compression routines and the GNU Multiple Pre- 
cision Arithmetic Library [9] for arithmetic. TFoRM uses the POSIX routines for multi- 
threaded processing and ParFoRM uses the MPI libraries for network communication. 

The preprocessor reads the input and edits this input according to the preprocessor 
syntax. Each time a statement is completed it is sent to the compiler. Whenever an end- 
of-module instruction is encountered the combined compiled code in the compiler buffers 
is executed. Once all active expressions have been processed with this code, the compiler 
buffer is emptied and the preprocessor continues reading the input. 

The compiler is offered complete statements and translates those first into an intermediate 
code which consists of so-called tokens. At this intermediate level some optimizations are 
performed. Next the tokens are translated into machine code for a virtual machine. The 
code is placed inside a compiler buffer. During this operation identical subexpressions are 
recognized to avoid storing them more than once. 

The pattern matcher is an extensive piece of code that matches the left hand sides of 
substitution statements, called patterns, with the contents of the terms in the expressions. 
Its most complicated part is dealing with wildcard variables and backtracking when there 
are several choices for matching and the choices made so far do not lead to a match. 

The center of the program is the terms generator. This is the core of the virtual machine. 
It controls the branching of the terms when substituting part of a term by one or more 
terms from either the compiler buffer, from other buffers (like tables, $-variables or other 
expressions), or from the automatic generation of terms as can happen during the expansion 
of special functions. 

The normalization routines bring terms to normal form. This is not completely trivial, 
because it also involves action to evaluate special functions. 

Finally, the sort routines bring collections of terms into a standard representation so that 
they form a unique expression. Its special features are that it is in principle disk based and 
can deal with very large expressions in a relatively short time. Because the representation 
of the terms is also very compact. Form can work with very large expressions. For TFoRM 
and ParFoRM the sorting system is extended, because with multiple workers involved the 
sorted results have to be combined into a single expression in the end. 

Form's main objects are expressions. An expression is a sequence of terms stored se- 
quentially on disk unless the expression is small enough to fit inside a cache buffer. The size 
of this buffer can be set by the user during startup. Each term is self-contained and con- 
tains no pointers. During execution no attempt is made to look for common subexpressions 
in different terms, hence there is no need for reference counts. In principle the terms are 
processed sequentially. This means that one term is taken from the input expression, the 
terms generator routine is called and it executes the first statement on it. The first term 
of the output of this statement is then processed by the generator routine with the second 
statement and so on. Figure [1] shows this process. 

After each statement the current term is normalized to make it ready for the next state- 
ment. When there are no more statements, the term is sent to the sorting system and the 
generator routine goes back one level, picks up the next term on that level, and then executes 
the next statement on it. This continues until the whole expansion tree has been traversed. 



S a,b,c,d; 
On HighFirst; 
L F = a+b+c+d; 
.sort 

Input expression second module: 



a+b+c+d 



id a = (a+b)'^2; 



id c = b+d; 



id b = b+1 ; 



.end 



^.^2 2ab 



3" 6" 



a'^2 2ab 



Sorting the terms: 



Output expression: 




4" 7f A8 11 



a'^2 2ab 2a b'^2 2b 1 



a'^2+2ab+2a+b'^2+4b+2d+3 



Figure 1: The tree expansion of terms in FoRM 



The sort routines act semi-independently; they accept terms, put them in buffers, sort the 
buffers when full, and, if needed, write information to disk. After the last term has been 
sent to the sorting routines, the sorting is finalized. If the expression is big, this may involve 
disk-to-disk sorting. As soon as the input is no longer needed, it is destroyed and the output 
of the current module becomes the input for the next module. 

Because all regular operations obtain their input from only a single term they are called 
local operations. Operations that take input from more than a single term are called non- 
local. Examples of non-local operations are the compares between terms during the sorting 
and the factorization of complete expressions. Form also has some semi-local operations. 
One of them is the collect statement. It places a predefined group of terms inside the 
argument of a function, which becomes part of a single term. Others involve the manipulation 
of functions of which one or more arguments contain expressions of more than a single term. 
When we will look at the polynomial operations we will see that they are inherently non- 
local although a large class of them can be treated as semi-local. The art of making good 
Form programs is often defined by how to construct inherently non-local operations by a 
combination of local operations and non-local sort instructions. We do not know of any 
systematic research of this. 

3 Open source and adding new features 

Starting with version 4, Form has become an open source program under the GNU General 
Public License ^J (GPL) version 3. The idea is to allow others to contribute to Form as 
well. Therefore it is important to describe shortly how new features can be added to it. 

Adding new functions is easiest. There is a header file (inivar.h) in which all built-in 
functions are defined and one needs to add one line there with the name of the function and 
its internal number which is, given as a name, defined in another header file (f types. h). 
Then, depending on the function, its action has to be programmed. This is usually done in 
the normalization routine. In complicated cases some special routines may have to be added 
that are called either from the normalization routine or from the generator routine. The 
source code provides enough examples to illustrate this process. 

Adding statements is often relatively simple in Form. The compiler has to recognize the 
keyword and send control to the routine that translates the statement. This routine has to be 
defined and it has to create the proper code for the virtual machine. The generator routine 
has to recognize this code and give control to the routine that executes the statement. Hence, 
apart from a few lines of code in other places only two routines have to be constructed. 

Adding extra features in the $-variables is similar, but requires more work. 

4 New features 

During the development of Form 3 many new features have been added. Most of those have 
been described before and are counted as belonging to version 3. However, the ones that 



were added during the later stages of version 3.3 are actually considered part of version 4 
and we will describe them here. For a complete overview of all features one should consult 
the reference manual. 

4.1 Polynomial factorization 

In Form there are three different occurrences of polynomials, namely as arguments of func- 
tions, as complete expressions, and as $- variables. These three cases are discussed separately. 

4.1.1 Function arguments 

When a polynomial is factorized, which is the argument of a function, the factors are returned 
as a sequence of arguments of the same function. The following program illustrates this. 

Symbols a,b,c; 

CFunction f; 

Format NoSpaces; 

Local F = f (3*(a+b)*(a+b+c)*(3*b+2*c)) ; 

Print "<1> It"; 

Fact Arg f ; 

Print "<2> tt" ; 

.end 
<1> +f (6*b*c"2+15*b"2*c+9*b"3+6*a*c"2+21*a*b*c+18*a*b"2+6*a"2*c+9*a"2*b) 
<2> +f (3,b+a,c+b+a,2*c+3*b) 

This use of the FactArg statement is different from what it used to be in version 3, when 
it would only find factors consisting of single terms. For backward compatibility we have 
provided the possibility to revert to the old behavior with the statement 

On OldFactArg; 
The new FactArg behavior is switched on again with 

Off OldFactArg; 
One may also specify the line 

OldFactArg ON 

in the setup file. This would be the easiest way to run old programs. 

Dealing with polynomials as function arguments has the drawback that neither the input 
polynomial nor the combined output polynomials can be larger than the maximum term 
size. 



4.1.2 Expressions 

Expressions do not liave the size constraint of function arguments, since they are disk based. 
However, their structure has no natural way to address the factors. To get around this we 
have selected a way that is in line with features of expressions: the bracket system. 

First we adopt the convention that whether an expression is in factorized or unfactorized 
form is a choice that is made at the output level. The following program illustrates this. 



Symbols 


a,b,c; 












CFunction f; 












Format 


Nospaces; 












Local F 


= 6*b*c~3+15*b"2*c' 


■2+9*b"3*c+6*a*c' 


~3 






+21*a*b*c' 


~2+18*a*l: 


)~2*c+6*a"2*c' 


~2+9*a' 


~2*b*c; 


Factorize F; 












Print ; 














.end 
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F= 

(3) 
*(c) 

*(c+b+a) 
*(2*c+3*b) 
*(b+a) ; 

The expression is first worked out as shown by the first statistics. Then the result is taken 
from the output and factorized. The factors are indicated by brackets in terms of the internal 
symbol f actor_. They are sorted as indicated by the second statistics. From this point on 
the expression is kept in factorized form, until it is unfactorized by using the statement 
Unf actorize. The internal representation of this factorized expression is 



F= 



factor.* (3) 
*factor_"2*(c) 
*f actor_"3* (c+b+a) 
*f actor_"4* (2*c+3*b) 
*factor_"5*(b+a) ; 



The writing routine does not write the special symbol f actor_, when the expression is in 
factorized form, but writes parentheses instead. It should be clear now that the substitution 

id a"2 = 1; 

will have no effect on the factorized expression, since the left hand side of the id statement 
is not present in it. 

When a substitution makes a factor zero, it is kept as a zero factor as shown by: 

Symbols a,b,c; 

CFunction f; 

Format Nospaces; 

Local F = 6*b*c~3+15*b"2*c"2+9*b"3*c+6*a*c~3 

+21*a*b*c"2+18*a*b~2*c+6*a"2*c~2+9*a"2*b*c; 
Factorize F; 

. sort 

id c = -b-a; 

Print ; 

.end 

F= 

(3) 
*(-b-a) 
*(0) 

*(b-2*a) 
*(b+a) ; 

Unfactorizing the expression will result in zero, of course. 

Unfactorize F; 

Print ; 

.end 

F=0; 

It is also possible to place factorized expressions directly into the input. For this the state- 
ments LocalFactorized and GlobalFactorized are present, which can be abbreviated as 
follows. 

Symbols a,b,c; 

CFunction f; 

LF F = 3*c*(a+b)*(a+b+c)*(3*b+2*c) ; 

Print ; 

.end 



F = 

( 3 ) 

* ( c ) 

* ( b + a ) 

* ( c + b + a ) 

* ( 2*c + 3*b ) ; 

Factors are recognized by the multiplication or division signs at ground level, as shown here. 

Symbols a,b,c; 

CFunction f; 

LP F = 3/2*(3/2)*c*((a+b)*(a+b+c))*(3*b+2*c); 

Print ; 

.end 



F = 



3 ) 

1/2 ) 

3/2 ) 

c ) 

b*c + b~2 + a*c + 2*a*b + a"2 ) 

2*c + 3*b ) ; 



Once we know that the factors are stored as brackets it is easy to refer to them. 

Local G = F[factor_~3] ; 

Print G; 

.end 

G = 
3/2; 

The number of factors in factorized expression can be obtained by the function NumFactors_. 

Local N = NumFactors_(F) ; 

Print N; 

.end 

N = 
6; 

A value of zero for the number of factors indicates that an expression is in the unfactorized 
state. 

Extra Bracket statements will have no effect on factorized expressions. Currently Form 
supports only one level of brackets and this level is taken by the factorization. The brackets 
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of the factorization use automatically the bracket index system to facilitate fast access to 
the brackets. 

If one factorizes an already factorized expression, the system will try to factorize the 
factors. In addition it combines constant terms and the factors are sorted. If there are 
factors of zero the result will be zero, unless 

Factorize (keepzero) F; 

is used, in which case the zeroes are combined into a single zero as the first factor. 

4.1.3 $- variables 

The third way of dealing with polynomials is as the content of $-variables. This has yet other 
requirements, because the $-variables have no access to the bracket system. In the case of 
the $-variables we also have to take into account that they can be manipulated both at the 
preprocessor and the execution level. At the same time we have to realize that, although 
$-variables are not restricted by the maximum term size, they do reside inside memory and 
therefore cannot be as large as regular expressions. 

When a $- variable has been factorized, we keep two copies of it: the unfactorized version 
and the factorized version. The factors remain in existence until the $-variable is redefined. 

The factorization is accomplished with the preprocessor instruction #FactDollar or with 
the statement FactDollar. The factors are accessed by giving their number between square 
brackets after the name of the $-variable as shown underneath. The zeroth element gives 
the number of factors. One may also use the NumFactors_ function to obtain the number of 
factors in a $- variable. Again a zero return value indicates that the variable has not been 
factorized. 

Symbols a,b,c; 

#$X = (a+b+c)*3*(a+b)*(3*b+2*c) ; 

#FactDollar $X; 

#Message The first factor of $X is '$X[1]' 
The first factor of $X is 3 

#Message The second factor of $X is '$X[2]' 
The second factor of $X is c+b+a 

#Message The third factor of $X is '$X[3]' 
~~~The third factor of $X is 2*c+3*b 

#Message The fourth factor of $X is '$X[4]' 
The fourth factor of $X is b+a 

#Message $X has '$X[0] ' factors 
~~~$X has 4 factors 

.end 

One may also use other $-variables or factors of $-variables between the brackets, provided 
they evaluate into valid factor numbers. 
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Symbols a,b,c,x; 

CFunction f ; 

Local F = f (3*(a+b)*(a+b+c)*(3*b+2*c)) 

+f (2*(a+c)*(a+b+c)) ; 
id f(x?$x) = 1; 
FactDollar $x; 
do $i = l,$x[0] ; 

Multiply f ($x[$i]) ; 
enddo ; 
Print +s; 
.end 

F = 

+ f (2)*f (c + a)*f(c + b + a) 

+ f(3)*f(b + a)*f(2*c + 3*b)*f(c + b + a) 

It should be noted that one cannot use constructions hke 

Multiply <f ($x [1] ) >* . . . *<f ($x [$x [0] ] ) > ; 

because the triple dot operator is a preprocessor facility and $x [0] is not defined until the 
execution phase. 

4.2 Rational polynomials 

To speed up the manipulation of certain types of expressions Form is equipped with the 
PolyFun mechanism. This has made programs like Mincer |10] considerably faster. There 
has been the need to extend this facility to dealing with rational polynomials. Therefore, 
the statement PolyRatFun is introduced in FoRM. It is declared in a similar way as the 
PolyFun: 

CFunction rat; 
PolyRatFun rat; 

The PolyFun and PolyRatFun declarations are mutually exclusive. The PolyRatFun is 
considered a special type of PolyFun and there can be only one of them at any moment. If 
one wants to switch back to a mode in which there is neither a PolyFun nor a PolyRatFun 
one can use 

PolyRatFun; 

to indicate that after this there is no function with that status. 

In order for the PolyRatFun to be effective, the function needs two arguments and the 
arguments should contain symbols only. The first argument is then interpreted as the nu- 
merator of a fraction and the second as the denominator of that fraction. If there is only 
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one argument it is seen as the numerator and FORM will add a second argument that has 
the value one. The use of more than two arguments results in an error. 

During the normalization of terms FORM will work out products of more than one 
PolyRatFun as in: 

Symbols x,y; 

CFunction rat; 

PolyRatFun rat ; 

Local F = rat(x,y)*rat(x+l,y+l) ; 

Print ; 

.end 

F = 

rat(x~2 + x,y"2 + y) ; 

and during the sorting FORM will add and subtract the fractions when the terms are other- 
wise identical as in 

Symbols x,y; 

CFunction rat; 

PolyRatFun rat ; 

Local F = rat(x,y)+rat(x+l,y+l) ; 

Print ; 

.end 

F = 

rat(2*x*y + x + y,y~2 + y) ; 

The fractions are at all times normalized. This means that the gcd of the numerator and 
the denominator has been divided out. 

Often the denominator of a PolyRatFun can be factorized. This should however not be 
done while the function is still a PolyRatFun, because more than two arguments will cause 
an error. The following example shows a way of doing it: 

Symbols x,j,xl,x2; 

CFunction rat,num,den; 

PolyRatFun rat ; 

Format NoSpaces; 

Local F = sum_(j , 1, 10,rat(l,x+j)) ; 

Print ; 

.sort 

F= 

rat(10*x"9+495*x"8+10560*x~7+127050*x~6+946638*x~5+4510275*x"4+ 
13667720*x"3+25228500*x~2+25507152*x+10628640,x~10+55*x-9+1320* 
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x~8+18150*x~7+157773*x"6+902055*x"5+3416930*x~4+8409500*x"3+ 
12753576*x"2+10628640*x+3628800); 

PolyRatFun; 

id rat(xl?,x2?) = num(xl)*den(x2) ; 

FactArg num den; 

ChainOut , num ; 

ChainOut , den ; 

id num(xl?number_) = xl; 

id nuin(xl?symbol_) = xl; 

id den(xl?number_) = 1/xl; 

id den(xl?symbol_) = 1/xl; 

Print ; 

.end 

F= 

num(lH-2*x)*num(966240+2143152*x+1903836*x~2+896368*x"3+247049* 
x"4+41140*x~5+4070*x"6+220*x~7+5*x~8)*den(H-x)*den(2+x)*den(3+x)* 
den(4+x)*den(5+x)*den(6+x)*den(7+x)*den(8+x)*den(9+x)*den(10+x) ; 

4.3 ToPolynomial, FromPolynomial and ExtraSymbols 

Sometimes it is better to have only symbols in an expression. This is for instance the case 
when an expression is put in the output for further processing in a C or Fortran program. 
The ToPolynomial statement converts all objects that are not symbols with positive powers 
into newly defined symbols and puts their definition in memory. The definitions of these 
symbols can also be printed. These symbols are called extra symbols and will have names 
that do not interfere with any user defined names. Alternatively the user has control over 
their names and can influence whether they are printed as individual or as array elements. 
This last option can be rather handy in strong typed languages as C. An example of these 
statements is: 

Symbols x,y,z; 

CFunction fl,f2; 

ExtraSymbols array Ab; 

Local F = fl(x+y)*z+f2(z)*(x+y) ; 

ToPolynomial; 

.sort 

Format C ; 

#write <example.c> " int Ab[-['extrasymbols_'+l}-] ,x,y,z;\n" 

#write <example.c> "IX" 

#write <example.c> " F = yoe",F 

.end 
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This program produces the following output in the file example . c: 
int Ab [3] ,x,y,z; 

Ab[l]=fl(y + x); 
Ab[2]=f2(z); 

F = z*Ab[l] + y*Ab[2] + x*Ab[2]; 



If the ExtraSymbols statement is omitted, the default names for the extra symbols are Zl_, 
Z2_, etc. If the Array option is omitted or the option Underscore is used the notation 
with the trailing underscore is used in the output, independent of the name selected. The 
statement FromPolynomial undoes the action of the ToPolynomial statement. 

New features like factorization are implemented internally in terms of symbols only. 
Therefore the routines that execute the ToPolynomial and FromPolynomial statements are 
used also internally for the factorization and in the future they may be used for more features. 

4.4 Div_, Reni_, Gcd_, Content, and Inverse. 

Now that Form has polynomial algebra implemented internally, there is need for several 
functions, that are available to users, that can do some basic operations. These are: 

Div_ The quotient of the division of the first by the second argument. The arguments can 
be polynomials. 

Rem_ The remainder of the division of the first by the second argument. The arguments 
can be polynomials. 

Gcd_ The gcd of all the arguments. The arguments can be polynomials. 

Content. The content of the argument, i.e., the argument divided by the gcd of all terms. 

Inverse. The inverse of the first argument Pi modulo the second argument P2, i-e., more 
explicitly, the polygon P3 such that (P3 • Pi) = 1 mod P2. 

Functions together with their arguments are usually restricted by the condition that they 
have to fit inside the maximum size of a term. With regular functions and an argument 
that is an expression, the expression is first expanded and hence subject to this restriction. 
The first four functions in this category however do not suffer from this limitation, because 
the expressions are only expanded during the operation itself. The same holds for when the 
arguments are $-variables. 
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4.5 Do loops 

The #Do instruction already exists since the original version of FoRM. This is a preprocessor 
feature that acts purely on the text of the input. Until recently its role at the execution 
level was taken over by the Repeat/EndRepeat construction, possibly in combination with If 
statements. With the advent of factorization and the need for organized access to the factors 
this became too complicated and we have implemented a proper Do/EndDo construction. The 
first question is of course what should serve as loop variable. For the original #Do instruction 
we use preprocessor variables, but we do not have these available at execution time. However, 
we do have the $-variables, which allows us to make the construction: 



'1 



Do $i = 1,5; 

id only x~$i = f (F[f actor_"$i] ) ; 
EndDo ; 

The boundary parameters in the Do statement should be either integers, that can be stored 
inside one Form word, or $-variables. They can also be $-variables with factor indicators 
as in 

Do $i = l,$e[0] ; 

Multiply f ($e[$i]) ; 
EndDo ; 

4.6 The transform statement 

There are several categories of manipulations of functions with either complicated arguments 
or many simple arguments that take enormous amounts of time when handled by means of 
the pattern matcher. An example is the following, where arguments of zero are replaced by 
ones and vice versa. 

Symbols x; 

CFunction f,g; 

Local F = f (1,0,1,0,0,1,0,1); 

Multiply g; 

repeat id g(?a)*f (x?,?b) = g(?a, l-x)*f (?b) ; 

id f*g(?a) = f(?a); 

Print ; 

.end 

F = 

f(0, 1,0, 1,1, 0,1,0); 

In this example the program has to use the repeat statement to go 8 times through the 
pattern matcher, plus a ninth time in the last id statement. The Transform statement is a 
whole group of operations in which the much simpler matching is done once and the rest is 
done internally by Form: 
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Symbols x; 

CFunction f,g; 

Local F = f (1,0,1,0,0,1,0,1); 

Transform f replaced, last) = (l, 0, 0, 1) ; 

Print ; 

.end 

F = 

f(0, 1,0, 1,1, 0,1,0); 

Basically, the Transform statement provides a syntax to deal with large groups of arguments. 
Each transformation consists of a subkey, indicating its type, followed by arguments that 
are enclosed by parentheses. Currently, there are 10 different subkeys. After that specific 
information may follow. The proper syntax of the subkeys is in the manual, but below 
follows a short description of what is available. The first transformations are quite generally 
applicable, while the latter are custom-made for a project on Multiple Zeta Values pTl[T2] . 

replace Like the replace, function, but allows also replacement of numbers. 

encode Combines a range of numerical arguments into a single number over a specified base 
number. 

decode The inverse of the encode subkey: a single argument is converted into a range of 
arguments. 

reverse Reverses a range of arguments. 

cycle Applies a cyclic permutation to a range of arguments. 

permute Permutes arguments according to a given permutation. 

tosumnotation There are two ways to characterize harmonic sums and harmonic poly- 
logarithms. In the sum notation the indices are non-zero integers and in the integral 
notation there are only the numbers -1,0,1. In the conversion from integral to sum 
notation a zero adds one to the absolute value of the nonzero number to the right of 
it as in 0,0,0,-1 -^ -4. 

tointegralnotation The inverse of tosumnotation, see example below. 

islyndon Tests whether the indicated range of arguments forms a Lyndon worqj according 
to the ordering of the arguments in FORM. A yes and no argument tell what the main 
term should be multiplied by, when the answer is yes or no respectively. 

tolyndon Will permute the given range in a cyclic manner until it is (if possible) a Lyndon 
word according to the ordering of the arguments in FoRM. Also here yes and no 
arguments should be specified. 



^One definition of a Lyndon word is the unique minimal cyclic permutation of a number of objects. 
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It is possible to use more than one subkey in a single transform statement. This improves 
efficiency. This is illustrated in an example from a program that manipulates Multiple Zeta 
Values. To start off with, the original version was: 

Symbol x,xl,x2; 

CFunction H,H1; 

Local F = H(3, 4, 2, 6, 1,1, 1,2); 

Print "<1> %t"; 

Repeat id H(?a,x? !{0, l},?b) = H(?a,0,x-l,?b) ; 

Print "<2> %t"; 

Multiply HI; 

Repeat id H(x?,?a)*Hl(?b) = H(?a)*Hl(?b, 1-x) ; 

id H*Hl(?a) = H(?a); 

Print "<3> It"; 

Repeat id H(xl?,x2?,?a) = H(2*xH-x2,?a) ; 

Print "<4> %t"; 

.end 
<1> + H(3,4,2,6,l,l,l,2) 

<2> + H(0, 0,1, 0,0, 0,1, 0,1, 0,0, 0,0, 0,1, 1,1, 1,0,1) 
<3> + H(l, 1,0, 1,1, 1,0, 1,0, 1,1, 1,1, 1,0, 0,0, 0,1,0) 
<4> + H(907202) 

In term of a single transform statement this becomes: 

CF H; 

L F = H(3, 4, 2, 6, 1,1, 1,2); 
Transform H tointegralnotation(l ,last) , 
replaced, last) = (0,1, 1,0) , 
encoded, last) :base=2; 
Print ; 
.end 

F = 

H(907202); 

This last version is faster, more readable, easier to program and less prone to errors. 

4.7 Random, and RanPerni_ 

Random, is a random number generator. It works according to an algorithm called the 
additive number generator as described in [13]. The used subscript pair is (38,89), which 
gives a longer cycle than the pair used in the reference. The function is called with an 
integer argument N. This integer has to be greater than one. The return value is a random 
number in the range [1, A^]. The generator can be initialized with the preprocessor instruction 
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#SetRandom <number>. If this instruction is never used the first use of the Random, function 
will use a standard initialization. It should be noticed that in TFoRM and ParFoRM each 
worker runs its own version of Random, and they are initialized differently, even though this 
is controlled by the #SetRandom instruction. One might argue that this makes the program 
give different results, depending on whether it is run with FoRM or with TFoRM, but that 
would be the case anyway, because the way the terms are distributed over the workers is 
non-deterministic. Hence, even if there were a unique sequence, the individual terms might 
not get the same number in the sequence. 

An additional useful function is the RanPerm_ function. Its first argument is the name 
of the output function. It generates a random permutation of the remaining arguments and 
puts this permutation in the output function. 

CFunction f; 

Local F = RanPerm_(f , 1, . . . ,6) ; 
Print ; 
.end 
F = 

f(3,l,6,2,4,5); 

These random functions can be helpful in testing and debugging programs among other 
things. 

4.8 FirstTerni_ 

The FirstTerm_ function needs a single argument that is the name of either an expression 
or a $-variable. It returns the first term in that object. This can be useful in a Gaussian 
elimination scheme or in a program that computes a Grobner basis. It is related to the 
FirstBracket_ function, which returns what is outside the first bracket of an expression 
that is in bracketed form. 

4.9 Prinie_, ExtEuclidean_ and MakeRationaL 

Several types of calculations can be performed far more efficiently when working modulo 
prime numbers. Typically, results modulo different prime numbers can be combined with 
the Chinese remainder theorem to construct solutions over the integers or rationals. For this 
purpose, the functions Prime_, ExtEuclidean_ and MakeRational_ are created. 

The function Prime_ generates prime numbers, starting from the maximum positive num- 
ber that fits inside a single Form word and working its way down. Numbers that are found 
are stored in a list, so that they do not have to be recomputed. The function Prime_(n) 
gives the n-th element of that list. If it did not exist yet. Form has to compute it. Notice 
that this way of dealing with prime numbers allows only for constrained values of n. The 
minimum value of a prime number that is allowed is determined by the maximum value that 
a power of a symbol can have in Form. On a 64-bits computer this allows for O{10^) prime 



numbers. One warning: the algorithm for prime number is not particularly fast and hence 
it might take of the order of a few hours to generate and store them all. 

In the extended Euclidean algorithm one does not only determine the greatest common 
divisor g of two numbers rii and n2, but also two numbers Xi and X2 such that g = Xini+X2n2- 
In the case that rii and n2 are relative prime (i.e., g = 1) xi and X2 are called the modular 
inverses of rii and n2 because Xiiii = 1 mod 722 and so is X2n2 mod rii. This algorithm 
is implemented in Form in the function ExtEuclidean_. It is useful for combining results 
modulo 111 and modulo n2 into a result modulo nin2 as the following example shows. 

#$pl = Prime. (1) ; 

#$p2 = Prime. (2) ; 

Symbols xl,x2,x3,x4; 

Off Statistics; 

Local F = 12345678901234567; 

. sort 

Local Gl = Mod2.(F,$pl); 

Local G2 = Mod2.(F,$p2) ; 

. sort 

#$cl = ExtEuclidean.($pl,$p2) ; 

#Inside $cl 

id ExtEuclidean.(xl?,x2?,x3?,x4?) = x2*x4; 
#Endlnside 

#$c2 = ExtEuclidean.($pl,$p2) ; 
#Inside $c2 

id ExtEuclidean.(xl?,x2?,x3?,x4?) = xl*x3; 
#Endlnside 
#$p3 = $pl*$p2; 
Modulus PlusMin '$p3'; 
Local H = Gl*$cl+G2*$c2; 
Print ; 

.end 

F = 

12345678901234567; 
Gl = 

- 229487668; 
G2 = 

- 183496428; 
H = 

12345678901234567; 

ExtEuclidean_ always returns integer numbers, while often calculations result in rational 
numbers. To obtain fractions after calculating modulo integers, the function MakeRational_ 
is available. It takes two arguments, which both are integer. The function is replaced by the 
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unique fraction of which both elements are less than the square root of the second argument 
and that, in calculus modulo this second number would give the same result as the first 
number modulo the second number. Example: 

#$in = prime_(l) ; 

#write <> "The prime number is °/o$",$m 
The prime number is 2147483587 

Local F = MakeRat ional_( 12345678, $m) ; 

Print ; 

.sort 

F = 

9719/38790; 

Modulus '$m'; 
Print ; 
.end 

F = 

12345678; 

4.10 Checkpoints 

There are various circumstances in which one can loose a large investment in computer 
time. One would be a power failure after many hours of running and another would be a 
syntax error near the end of a program. Whereas the second case is one the user can do 
something about by first testing the program on small examples, the first case is usually not 
under control. To avoid the potential loss of big computer resource investments. Form is 
equipped with a user controlled backup mechanism. At the end of a module the user has 
the possibility to make a snapshot of the current state. If, for one reason or another, the 
program fails at a later stage, it can be restarted from the last snapshot. This snapshot is 
stored in a disk file and contains the contents of all relevant files and all relevant contents 
of memory locations. It is tied to a given executable file of Form, TForm or ParFoRM 
and only meant to continue execution when the failure condition has been resolved. One 
should not use it as a practical means of calculation backup. One should also not change 
the number of workers in TFoRM or ParFoRM. FoRM remembers the reading position in 
open input files. Hence, altering such a file before this position and changing its length may 
have disastrous consequences. Changing it after the current reading position should give no 
problems. This would be a way to repair an error that leads to a crash in the final stages of 
a program. 

The checkpoint feature is activated and deactivated with the statements: 

On Checkpoint [<OPTIONS>] ; 
Off Checkpoint; 
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The statement On Checkpoint will cause a snapshot to be made at the end of the module 
and each module that follows until a statement Off Checkpoint is encountered. If the 
checkpoints are already active and a new On Checkpoint statement is issued, the options of 
the new statement override the corresponding options of the old statement. 
An option of the form 

<NUMBER> [<UNITS>] 

specifies a time interval and tells Form to do the writing at module ends only if more time 
than this given time has elapsed since the last recovery data write. The units are optional 
and default to seconds. Possible units are: s for seconds, m for minutes, h for hours, and d 
for days. 

Form can run external programs before and after the recovery data writing. The options 
to specify the filename of such a program are: 

runbef ore= " <FILENAME> " 
runaf t er= " <FILENAME> " 
run="<FILENAME>" 

The option run specifies the filename for both cases simultaneously. <FILENAME> needs to 
be a valid filename referring to a proper executable. Depending on the operating system 
this can mean the file needs to have the correct access permissions and to lie in the current 
search path. The programs will only be run if the condition on the time interval has been 
fulfilled. This can be used to check for enough disk space, and to compress or move the 
backups if desired. 

The return value of the program run before the data writing will be interpreted. If it 
is a value unequal to zero or if the execution itself returns an error (e.g., if the executable 
cannot be found), no data will be written at this module's end. Independently, the program 
specified to be run after the data writing will always be executed. 

The -R option in the calling of Form invokes the recovery procedure after a crash. 
Imagine that the original program was run with the command 

form -S my. set long.frm 

and it crashed, then 

form -S my. set -R long.frm 

will continue its execution at the point of the last snapshot. 

The writing of the recovery data proceeds in several steps. First, all the data inside Form 
is written to an intermediate file. Second, existing files like a store or hide file are copied. At 
this point, the old recovery data basically gets overwritten. Finally, the intermediate data 
file is renamed into the proper recovery file. If any problem has occurred, it is signaled to 
the user. If during the copying of the store, hide, or scratch file anything goes wrong, the old 
recovery state will be overwritten and left in an unusable, undefined state. An improvement 
of the situation would be to copy all files first to intermediate files and finally rename them, 
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but it would triple the amount of space needed on the disk. Because these files can be very 
large, this was considered impractical. The option runbef ore can also be used to copy the 
backup first. 

4.11 System independent save files 

New is also the portability of the save files (files created with the Save statement and 
recovered with the Load statement) between computers with a different architecture. Of 
course the system does not work in 100% of the cases, because on a computer with a 64- 
bits architecture one may define many more variables than on a computer with a 32-bits 
architecture. If there are too many variables, the computer with the 32-bits architecture 
will give a nametable overflow when reading expressions with too many variables. Similarly 
there are different restrictions on maximum powers of symbols. The implementation of this 
portability made it necessary to redefine the format of the save files. Hence old save files 
will not work with version 4. The best way to convert old save files is by printing their 
contents to regular text output with the old version of Form and then reading the resulting 
expression with a new version of Form that can put it in a new save file. 

5 Polynomial algorithms 

For the manipulation of polynomials and the calculation of gcds and factorizations. Form 
uses a number of well-known algorithms. To store the polynomials, a degree sparse and 
variable dense representation is used [H]. Degree sparse indicates that only non-zero terms 
are stored, so that the representation of sparse polynomials is efficient. Variable dense 
indicates that for each monomial a vector with the exponents of all variables is stored. This 
is slightly inefficient regarding storage, but typically speeds up hard polynomial computations 
such as factorization. For some algorithms, dense univariate polynomials are stored as an 
array of coefficients if that increases the efficiency. 

Polynomial addition and subtraction is implemented by merging two polynomials. Mul- 
tiplication and division is performed by using a binary heap of monomials to find the next 
term quickly and achieve good performance [15]. All polynomial operations can be per- 
formed over the integers, modulo a prime number or modulo powers of primes. The latter 
are used for intermediate results in gcd computations and factorization. The prime powers 
are cached in a table for efficiency. 

To calculate the gcd of two univariate polynomials, Euclid's algorithm is used [IB] . For 
the multivariate case Form tries a heuristic algorithm a couple of times [17] , which usually 
results in the answer. Upon failure, Zippel's sparse modular algorithm is used [18]. This 
algorithm calculates univariate gcds modulo prime numbers and constructs from them the 
multivariate gcd with polynomial interpolation and the Chinese remainder algorithm. 

For the factorization of univariate polynomials modulo prime numbers Berlekamp's algo- 
rithm is used [19] . To obtain the factorization over the integers Hensel lifting is employed [20] . 
Eventual spurious factors are combined by a brute force approach. Multivariate polynomials 
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are reduced to univariate ones by substituting appropriate integers for all but one variable, 
so that Berlekamp's algorithm can be used. The multivariate factorization is then con- 
structed with the multivariate generalization of Hensel lifting. To circumvent as much of 
the multivariate lifting as possible, coefficients are predetermined by equating the factoriza- 
tion and the original polynomial [21]. The leading coefficient problem is solved by Wang's 
algorithm [2T] . 

A caching mechanism speeds up many realistic calculations in which the program at- 
tempts to factorize the same polynomials more than once. 

6 Parallel versions 

The fact that an expression in Form is a sequence of self-contained terms suggests that 
they can be processed in parallel for local operations. Form has two versions that allow the 
use of more than a single processor: TFoRM [6] and ParFoRM [5J. TFoRM uses multiple 
threads in a single multicore computer in which the cores share the memory. ParFORM 
uses the MPI library to establish communication between different processes, which do not 
share the memory and may be on different computers, and to distribute the calculation over 
these processes. In principle both programs are supposed to work in the same way from the 
viewpoint of the user. In practice it was much easier to implement TFoRM, because one 
does not need to send information, each time some new data is needed. The cases that one 
needs to send data via the MPI on ParFoRM include: redefine statements, $- variables, 
expressions appearing in right hand sides of definition or substitution statements, and the 
global table for converting the extra symbols. 

Both the parallel versions have been described before, but ParFoRM has always been 
rather incomplete in the sense that some facilities were either missing or not working properly. 
With version 4 this has been fixed. It should now be possible to run all TFORM programs 
also with ParFoRM. Of course also all regular sequential Form programs should run both 
with TFoRM and ParFoRM. 

The two programs have different benefits. TFoRM needs far less communication and 
makes good use of several cores, but it suffers from the presence of only a single disk. 
ParFoRM needs much more communication, but different computers usually have their own 
disk and hence there is less chance of slowdown due to traffic jams at the disk(s). It is up 
to the user to decide what is best for a given problem. 

Both versions will suffer from bottlenecks. Some parts of the structure of Form cannot 
be parallelized in principle or are hard to parallelize. An example is the final sorting on the 
master. Hence there will be not much further improvement in the performance beyond the 
use of a given, problem dependent, number of processors. This is a problem that is worked 
on. 
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7 Facilities 

With the upgrade to the new version also the manual has been extended. The complete 
manual comes in 4 versions: 

.tex the original LaTeX sources, 

.ps the processed postscript file, 

.pdf the processed PDF file, and 

.html the online version. 

This manual is a reference manual and should not be confused with a tutorial. The old 
tutorial by A. Heck and collaborators is still a very good introduction to Form, but it 
suffers from the fact that it was written for version 2. After version 2 the syntax has 
changed somewhat and also many new features were added. One of the future projects is to 
convert this tutorial to the syntax of version 4. 

To make the open source aspect of Form more realistic new documentation has been 
written. It explains about the inner working of Form and many of its routines. Most 
routines have a header that is compatible with the documentation system Doxygen. This 
should help with finding ones way through the sources. 

In addition a system has been programmed in the language Ruby to allow certification 
of new executables. In this system many examples of Form programs are stored, together 
with the crucial parts of their output. Making a certification run will test all these examples 
and whether they still give the correct answer. It is the intention to add more examples in 
the future. 

To facilitate communication between users or between users and authors a forum has 
been created. Here people can report on new Form programs, problems with installation or 
execution, request new features, etc. We had to protect the forum a little bit, because at a 
given moment there were large numbers of spam attacks. Users should sign up and answer 
an easy question, before they are admitted and can make posts. Currently the forum can be 
found at ,http : / /www . ni khef ■nl/~f orm/f orum, . 

The Form program is all original code in the C and C++ languages. It uses a few 
generally available libraries. These are libgmp, librt, zlib, the POSIX multithread system, 
the MPI library, and, of course, the standard C and C++ libraries. 

The GMP (GNU Multi Precision Arithmetic Library) deals with the arithmetic of very 
large numbers. Actually Form uses only the multiplication, the division and the gcd rou- 
tines (all for integers) from this library. If the GMP library is not present, FoRM has its 
own routines which may be slower, because in the GMP library there are some assembler 
statements that do not have an equally efficient equivalent in the C language. In addition 
the GMP library uses better algorithms for extremely large numbers. The zlib is used for 
data (de) compression in the sort file (only if the user requests this) and for the tablebase 
facihty. Form can be built without it. The POSIX library is used for TFoRM. Without it 
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one cannot build TFoRM. The MPI library is used for ParFoRM and without it one cannot 
build ParFoRM. 



8 Conclusions 

The new version 4 brings a giant leap forward in the capabilities of Form. The facility 
that people missed most, factorization, has been added. Many new commands and functions 
pave the way for future calculations that apply significantly different algorithms from those 
used in the past. And if these features are not sufficient, users may implement their own 
additions. 
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